Setting general connection properties

This section describes how to configure general connection properties. For an explanation of how to configure advanced connection properties, see Setting advanced connection properties.

To set the general connection properties:

1. Click the Manage Endpoint Connections toolbar button.

   The Manage Endpoints Connections dialog box opens.

2. Click the New Endpoint Connection toolbar button.

   The Name, Description, Type and Role fields are displayed on the right.

3. In the Name field, specify a display name for the endpoint.
4. In the Description field, optionally type a description for the Confluent Cloud endpoint.
5. Select Target as the endpoint Role.

6. Select Confluent Cloud as the endpoint Type.

   The dialog box is divided into General and Advanced tabs.

7. Continue as described below.

Kafka access

In the Kafka Access section, the following options can be set:

• Bootstrap Server: The bootstrap server.

  Example:

  pkc-159m3.us-east4.gcp.confluent.cloud:9092

• Authentication

  • API key (SASL/PLAIN)

    • API Key: The Confluent Cloud API key required to access the Kafka cluster.

    • API Secret: The Confluent Cloud API secret required to access the Kafka cluster.

  • OAuth (SASL/OAUTHBEARER)

    • Token endpoint URL: The OAuth 2.0/OIDC identity provider's endpoint for refreshing or accessing tokens.

      Example:

      https://example.com/oauth2/v1/token

    • Client ID: The client ID for authenticating against the OAuth authorization server.

    • Client secret: The client secret for authenticating against the OAuth authorization server.

    • Scope: This is optional. Use to restrict the permissions of the requesting application (Replicate).

    • Logical cluster ID: The unique identifier used to namespace and isolate resources belonging to different tenants or users.

      Example:

      lkc-123456

    • Identity pool ID: This is optional. The unique identifier used to map external OAuth/OIDC identities (from Azure AD, Okta, and so on.) to specific Kafka permissions.

Message properties

In the Message Properties section, set the following properties:

• Format: JSON or Avro.
• Compression: None (the default), gzip, or Snappy.

Data message publishing

• In the Publish data to field, choose one of the following:
  • Specific topic: Select this option to publish the data to a single topic. Then, in the Topic field, either type a topic name or use the browse button to select a topic.

  • Separate topic for each table: Select this option to publish the data to multiple topics corresponding to the source table names.

    The target topic name consists of the source schema name and the source table name, separated by a period (e.g. "dbo.Employees").

• From the Partition strategy drop-down list, field, select either Random or By message key. If you select Random, each message will be written to a randomly selected partition. If you select By message key, messages will be written to partitions based on the selected By message key (described below).

• From the Message key drop-down list, field, select one of the following:

  • None - To create messages without a message key.

  • Schema and table name - For each message, the message key will contain a combination of schema and table name (e.g. "dbo+Employees").

    When By message key is selected as the Partition strategy, messages consisting of the same schema and table name will be written to the same partition.

  • Primary key columns - For each message, the message key will contain the value of the primary key column.

    When By message key is selected as the Partition strategy, messages consisting of the same primary key value will be written to the same partition.

Schema Registry

In the Schema Registry section, you can configure the Schema Registry connection and subject properties.

Connection properties

• Schema Registry Server: The Schema Registry server.

  Example:  

  psrc-lx2rz.us-east4.gcp.confluent.cloud:443

• Authentication

  • API key (SASL/PLAIN)

    • API Key: The Confluent Cloud API key required to access the Kafka cluster.

    • API Secret: The Confluent Cloud API secret required to access the Kafka cluster.

  • OAuth (SASL/OAUTHBEARER)

    • Token endpoint URL: The OAuth 2.0/OIDC identity provider's endpoint for refreshing or accessing tokens.

      Example:

      https://example.com/oauth2/v1/token

    • Client ID: The client ID for authenticating against the OAuth authorization server.

    • Client secret: The client secret for authenticating against the OAuth authorization server.

    • Scope: This is optional. Use to restrict the permissions of the requesting application (Replicate).

    • Logical cluster ID: The unique identifier used to namespace and isolate resources belonging to different tenants or users.

      Example:

      lkc-123456

    • Identity pool ID: The unique identifier used to map external OAuth/OIDC identities (from Azure AD, Okta, and so on.) to specific Kafka permissions.

• Use proxy server - Select to publish to the Schema Registry via a proxy server.

  • Proxy connection: Choose one of the following:
    • Use endpoint proxy setting: Choose this option to use the proxy settings defined for this endpoint.
    • Use default proxy settings: Choose this option to use the default proxy settings defined for all endpoints. The default settings are defined in the Replicate server settings' Default Proxy Server tab.
  • Host name - The host name of the proxy server.
  • Port - The port via which to access the proxy server.
  • User name - The user name for connecting to the proxy server.
  • Password - The password for connecting to the proxy server.
  • Scheme - Select which protocol to use to access the server (HTTP or HTTPS).
  • SSL CA Path - When HTTPS is the selected Scheme, either enter the path to the CA key file on the Replicate machine, or use the Browse button to upload the file contents into the endpoint settings. Using the Browse button eliminates the need to access files on the Replicate machine, which is recommended for better security. If you use the Browse button, the file content will be stored as base64-encoded data in this field.

Subject properties

• From the Subject name strategy drop-down list, optionally change the default subject name strategy:

  • Schema and Table Name (Default)

  • Topic Name

  • Record Name

  • Topic and Record Name - See also: Message format.

  Information note

  • The default strategy (Schema and Table Name) is a proprietary Qlik strategy while the other three are standard Confluent subject name strategies.

  • If you select Topic Name or Topic and Record Name, be aware of the following considerations:

    • When the "add $topic column" method is used, the subject will be created only once (as the $topic expression might create multiple subjects).

      See also steps 3 and 4 in Overriding the default settings.

    • The "Metadata only" Advanced run option is not supported. This is because Replicate depends on the arrival of the first record per table in order to create the subject.

  For more information on Confluent's subject name strategies, see https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy

• From the Subject compatibility mode drop-down list, optionally change the default compatibility mode. The default is Use Schema Registry defaults.

  For an explanation of the available modes, see https://docs.confluent.io/platform/current/schema-registry/avro.html#compatibility-types

  Information note

  When publishing messages to Confluent Schema Registry, the default subject compatibility mode for all newly created Control Table subjects will be None, regardless of the selected Subject compatibility mode.

  Should you wish the selected Subject compatibility mode to apply to Control Tables as well, set the setNonCompatibilityForControlTables internal parameter to false.